notebook.community

Edit and run

Example paths from THREDDS Catalog Services

All of the relative path structures found to date and how to normalize those (if possible).

Note that this is derived from the catalog.xml responses and does not refer to the HTML representations of OPEnDAP or THREDDS servers.

In many cases, the lower-level responses have been truncated.

I've limited the examples to highlight certain issues and patterns and it is not meant to be (and not going to be) a complete "State of THREDDS responses" document.

Findings

The main issue from a machine-readable access point of view is in the differences between the HTML representation (from OPeNDAP Hyrax) and the catalog service (THREDDS) responses. In this example, we have the HTML page and the catalog.xml response for the same object:

The HTML View

The Catalog View

Many of the links for the listed datasets are not available through the catalog response (OPeNDAP/Hyrax v THREDDS, sure, but the TDS XML link is included in small print at the bottom of each HTML view). If we approach that from a harvester point of view, we have the resources fetched through the HTML and a different set, assuming we can correctly build the URLs, from the TDS XML. For custom or service-specific harvesters, a scenario in which only one of the resources is harvested is not out of the question.

I have not, to date, found anything that handled catalog services relying on query parameters that either provided those paths or some way of generating the paths. All of the iteration solutions rely on route changes and not query parameters. Somewhat similar, for catalogs that reference a WxS service option, I have not found a reliable pattern for generating the URLs or for understanding if a service is supported for a dataset. They are often "global" in the catalog.xml and I'm not sure if the service errors indicate an unsupported option for the dataset/granule or there's an issue with the actual service or I am not using the correct URL structure.

Supported Service Options

  • simple (unnested) or compound (nested with reference to the parent at the leaf dataset)
  • services are for the most part defined in each catalog response regardless of whether they relate to elements within
  • in some cases, the individual services in a compound element may not function for a given dataset leaf file (specifically, some WxS services are not supported but it's unclear if that is at the dataset or server level and the serviceName for a dataset is not that fine-grained).
  • some service @base values do not match the structure in other elements, ie can be 'opendap/hyrax' but the URL will resolve with just 'opendap'.

Catalog Ref Elements

  • there are two patterns for the href relative paths (ignoring the full path values of the UCAR/EOL v1.0.2) related to the catalogRef's parent element. If the catalogRef is a child of the root or of another catalogRef, the relative path begins from the host. If it's the child of a dataset element, the relative path begins at the dataset parent level (drop the terminal catalog.xml and append the href path). There's no path information at the parent dataset level that is stable (the name and ID attributes can sometimes reflect the path structure but there's no guarantee).
  • name elements are often empty strings and may be left out, rendering the response invalid.
  • it appears not expressive enough for some uses (see the EOL examples) where the @title value includes a token describing the relationship of the child catalogRef to the parent leaf node.

Dataset Elements

  • Two basic structures - a parent dataset with no @urlPath or data-specific children (no date, no dataSize, no access elements) but with some number of catalogRef or dataset elements.
  • may have one or more access points - if using access elements, can be multiple instances referring to a defined service (access/@serviceName) but if dataset/@serviceName, only the one service is supported.
  • this is the only level where the service definitions matter. If @serviceName, generate the URL based on the host, the intermediate service @base relative path and the leaf's urlPath.
  • can't rely on the date or dataSize elements to determine the leaf dataset. The only reliable way (via XPath considerations) is to look for the urlPath either as an attribute or in a child access element. Note that the leaf can still contain catalogRef children.

Metadata Elements

  • often, these do not have an explicit endpoint.
  • or text.

THREDDS (or hyrax or OPEnDAP, I don't know offhand what generates these other than "UnidataDD2MI.xsl") ISO records don't contain the fileIdentifier or MD_DataIdentification/CI_Citation/identifier values. An example response:

https://rsg.pml.ac.uk/thredds/iso/cnr/3b42-3h/1998/01/01/3B42.19980101.00.7.nc

(Secondary bug - the dimension units href values are never valid - www.example.com... URIs.)

The identifier issue (and general presence of metadata values in the ISO running through the THREDDS extension) is related to how the NcML is generated first (no keywords or abstract here, no keywords or abstract in the ISO). The fileIdentifier/citation identifier is related to the UnidataDD2MI.xsl code.

For services providing full paths and not following the custom, there does not appear to be a straightforward way to iterate up the catalog tree from a child node.

1. From Version 1.0.2

The tree.

http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.thredds.xml

a dataset parent containing many catalogRef children

no service definition

each metadata or catalogRef element contains a full URL in @href

<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
    xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0 http://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.0.2.xsd"
    version="1.0.2" name="EOL data server archive">
    <dataset authority="edu.ucar.eol.codiac" ID="ucar.ncar.eol" name="EOL projects" harvest="true">
        <metadata xlink:href="http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.metadata.xml"
            metadataType="THREDDS" inherited="false"/>
        <catalogRef
            xlink:href="http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.3CPO.thredds.xml"
            xlink:title="3CPO: 3CPO Trace Gas Measurements Program"/>
        <catalogRef
            xlink:href="http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.ABLE_2B.thredds.xml"
            xlink:title="ABLE-2B: Amazon Boundary Layer Experiment - 2B"/>
        <catalogRef
            xlink:href="http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.ABLE_3B.thredds.xml"
            xlink:title="ABLE-3B: Arctic Boundary Layer Expedition"/>
    </dataset>
</catalog>

http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.ARC_MIP.thredds.xml

a dataset parent containing many catalogRef children

no service definition

each metadata or catalogRef element contains a full URL in @href

note the shift from .project. to .dataset. in the file path (this doesn't match the other patterns and, more to the point, doesn't contain the project reference (do not suggest parsing the title, this is not a reliable solution)).

<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
    xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0 http://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.0.2.xsd"
    version="1.0.2" name="ARC-MIP: Arctic Regional Climate Model Intercomparison Project">
    <dataset authority="edu.ucar.eol.codiac" ID="ucar.ncar.eol.project.ARC_MIP"
        name="ARC-MIP: Arctic Regional Climate Model Intercomparison Project" harvest="true">
        <metadata
            xlink:href="http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.project.ARC_MIP.metadata.xml"
            metadataType="THREDDS" inherited="false"/>
        <catalogRef
            xlink:href="http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.dataset.61_001.thredds.xml"
            xlink:title="ARCMIP: Atmospheric Initial and Boundary Conditions (Cassano)"/>
        <catalogRef
            xlink:href="http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.dataset.61_002.thredds.xml"
            xlink:title="ARCMIP: CASPR-ARCMIP 1997 Dataset (Curry)"/>
    </dataset>
</catalog>

http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.dataset.61_006.thredds.xml

a dataset leaf node, relative path in access/@urlPath

a single service element, related to access/@serviceName

actual data access through service/@serviceBase + access/@urlPath (http://data.eol.ucar.edu/codiac/fgr_form/id=61.006)

<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
    xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0 http://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.0.2.xsd"
    version="1.0.2" name="ARCMIP: EASE-Grid Model Data">
    <service name="codiac_order" base="http://data.eol.ucar.edu/codiac/fgr_form/id="
        serviceType="WebForm"
        desc="Codiac access: order, subset, change formats; pickup data by FTP"/>
    <dataset authority="edu.ucar.eol.codiac" ID="ucar.ncar.eol.dataset.61_006"
        name="ARCMIP: EASE-Grid Model Data" harvest="true">
        <metadata
            xlink:href="http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.dataset.61_006.metadata.xml"
            metadataType="THREDDS" inherited="true"/>
        <access serviceName="codiac_order" urlPath="61.006"/>
    </dataset>
</catalog>

http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.dataset.106_270.thredds.xml

(second dataset leaf node example)

a dataset leaf node, relative path in access/@urlPath

contains some related content in a catalogRef child node (relationship identified in the @title as "Companion: ....")

actual data access through service/@serviceBase + access/@urlPath

<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
    xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0 http://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.0.2.xsd"
    version="1.0.2"
    name="ArcticRIMS: Daily Precipitation From Statistical Reconstructions , Aggregated by Administrative Regions">
    <service name="codiac_order" base="http://data.eol.ucar.edu/codiac/fgr_form/id="
        serviceType="WebForm"
        desc="Codiac access: order, subset, change formats; pickup data by FTP"/>
    <dataset authority="edu.ucar.eol.codiac" ID="ucar.ncar.eol.dataset.106_270"
        name="ArcticRIMS: Daily Precipitation From Statistical Reconstructions , Aggregated by Administrative Regions"
        harvest="true">
        <metadata
            xlink:href="http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.dataset.106_270.metadata.xml"
            metadataType="THREDDS" inherited="true"/>
        <access serviceName="codiac_order" urlPath="106.270"/>
        <catalogRef
            xlink:title="companion: ArcticRIMS: A Regional, Integrated Hydrological Monitoring System for the Pan-Arctic Landmass"
            xlink:href="http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.dataset.106_ARCSS161.thredds.xml"
        />
    </dataset>
</catalog>

This set includes a reference:

http://data.eol.ucar.edu/jedi/catalog/ucar.ncar.eol.dataset.106_224.metadata.xml

to a UNIDATA metadata record.

2. A version 1.0.1 example from NOAA

URL: http://www.esrl.noaa.gov/psd/thredds/catalog.xml

The tree.

http://www.esrl.noaa.gov/psd/thredds/catalog.xml

the parent node

nested service nodes with relative path bases (ex: "/psd/thredds/dodsC/")

catalogRefs with relative path URLs as @href (ex: "/psd/thredds/catalog/Datasets/catalog.xml")

rel paths start at root node (INTERSECTS)

note the use of "catalog" in the rel paths but the lack of "catalog" as a service reference


<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" name="THREDDS PSD Test Catalog" version="1.0.1">
  <service name="all" serviceType="Compound" base="">
    <service name="odap" serviceType="OPENDAP" base="/psd/thredds/dodsC/" />
    <service name="http" serviceType="HTTPServer" base="/psd/thredds/fileServer/" />
    <service name="wcs" serviceType="WCS" base="/psd/thredds/wcs/" />
    <service name="wms" serviceType="WMS" base="/psd/thredds/wms/" />
  </service>
  <catalogRef name="" xlink:href="/psd/thredds/catalog/Datasets/catalog.xml" xlink:title="Datasets">
    <metadata inherited="true">
      <serviceName>all</serviceName>
      <dataType>GRID</dataType>
    </metadata>
    <property name="DatasetScan" value="true" />
  </catalogRef>
  <catalogRef xlink:href="aggregations.xml" xlink:title="Aggregations" name="" />
</catalog>

http://www.esrl.noaa.gov/psd/thredds/catalog/Datasets/catalog.xml

nested service nodes with relative path bases (ex: "/psd/thredds/dodsC/")

dataset parent with catalogRef children

relative paths, non-overlapping at catalogRef/@href

<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.0.1">
  <service name="all" serviceType="Compound" base="">
    <service name="odap" serviceType="OPENDAP" base="/psd/thredds/dodsC/" />
    <service name="http" serviceType="HTTPServer" base="/psd/thredds/fileServer/" />
    <service name="wcs" serviceType="WCS" base="/psd/thredds/wcs/" />
    <service name="wms" serviceType="WMS" base="/psd/thredds/wms/" />
  </service>
  <dataset name="Datasets" ID="Datasets">
    <metadata inherited="true">
      <serviceName>all</serviceName>
      <dataType>GRID</dataType>
    </metadata>
    <catalogRef xlink:href="20thC_ReanV2/catalog.xml" xlink:title="20thC_ReanV2" ID="Datasets/20thC_ReanV2" name="" />
    <catalogRef xlink:href="COBE/catalog.xml" xlink:title="COBE" ID="Datasets/COBE" name="" />
    <catalogRef xlink:href="COBE2/catalog.xml" xlink:title="COBE2" ID="Datasets/COBE2" name="" />
    <catalogRef xlink:href="CarbonTracker/catalog.xml" xlink:title="CarbonTracker" ID="Datasets/CarbonTracker" name="" />
    <catalogRef xlink:href="NARR/catalog.xml" xlink:title="NARR" ID="Datasets/NARR" name="" />
    <catalogRef xlink:href="Timeseries/catalog.xml" xlink:title="Timeseries" ID="Datasets/Timeseries" name="" />
    <catalogRef xlink:href="cmap/catalog.xml" xlink:title="cmap" ID="Datasets/cmap" name="" />
    <catalogRef xlink:href="coads/catalog.xml" xlink:title="coads" ID="Datasets/coads" name="" />
  </dataset>
</catalog>

http://www.esrl.noaa.gov/psd/thredds/catalog/Datasets/noaa.oisst.v2.derived/catalog.xml

nested service nodes with relative path bases (ex: "/psd/thredds/dodsC/")

dataset parent leaf dataset children

relative paths at dataset/@urlPath (ex: "Datasets/noaa.oisst.v2.derived/sst.day.ltm.nc")

this one, at least, has a valid wxs service: http://www.esrl.noaa.gov/psd/thredds/wcs/Datasets/noaa.oisst.v2.derived/sst.day.ltm.nc?service=wcs&version=1.1.0&request=getcapabilities

<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.0.1">
  <service name="all" serviceType="Compound" base="">
    <service name="odap" serviceType="OPENDAP" base="/psd/thredds/dodsC/" />
    <service name="http" serviceType="HTTPServer" base="/psd/thredds/fileServer/" />
    <service name="wcs" serviceType="WCS" base="/psd/thredds/wcs/" />
    <service name="wms" serviceType="WMS" base="/psd/thredds/wms/" />
  </service>
  <dataset name="noaa.oisst.v2.derived" ID="Datasets/noaa.oisst.v2.derived">
    <metadata inherited="true">
      <serviceName>all</serviceName>
      <dataType>GRID</dataType>
    </metadata>
    <dataset name="sst.day.1981-2010.ltm.nc" ID="Datasets/noaa.oisst.v2.derived/sst.day.1981-2010.ltm.nc" urlPath="Datasets/noaa.oisst.v2.derived/sst.day.1981-2010.ltm.nc">
      <dataSize units="Mbytes">47.31</dataSize>
      <date type="modified">2011-09-09T16:18:50Z</date>
    </dataset>
    <dataset name="sst.day.ltm.nc" ID="Datasets/noaa.oisst.v2.derived/sst.day.ltm.nc" urlPath="Datasets/noaa.oisst.v2.derived/sst.day.ltm.nc">
      <dataSize units="Mbytes">94.61</dataSize>
      <date type="modified">2010-11-14T22:32:31Z</date>
    </dataset>
  </dataset>
</catalog>

3. version 1.0 example

URL: http://acdisc.gsfc.nasa.gov/opendap/hyrax/catalog.xml

The tree.

http://acdisc.gsfc.nasa.gov/opendap/hyrax/catalog.xml

unnested set of services (ex: "/opendap/hyrax")

although there is that catalogRef at the end pointing to an ncml catalog (sometimes described in the nested service block)

catalogRef @href can be converted using urljoin (strip catalog.xml and append href)

<?xml version="1.0" encoding="UTF-8"?>
<thredds:catalog xmlns:fn="http://www.w3.org/2005/02/xpath-functions"
             xmlns:thredds="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
             xmlns:xlink="http://www.w3.org/1999/xlink"
             xmlns:bes="http://xml.opendap.org/ns/bes/1.0#">
<thredds:service name="dap" serviceType="OPeNDAP" base="/opendap/hyrax"/>
<thredds:service name="file" serviceType="HTTPServer" base="/opendap/hyrax"/>
    <thredds:dataset name="/" ID="/opendap/hyrax/">
        <thredds:catalogRef name="Aqua_AIRS_Level3" xlink:href="Aqua_AIRS_Level3/catalog.xml"
                      xlink:title="Aqua_AIRS_Level3"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Aqua_AIRS_Level3/"/>
        <thredds:catalogRef name="Aqua_MODIS_Level3" xlink:href="Aqua_MODIS_Level3/catalog.xml"
                      xlink:title="Aqua_MODIS_Level3"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Aqua_MODIS_Level3/"/>
        <thredds:catalogRef name="Catalog" xlink:href="Catalog/catalog.xml" xlink:title="Catalog"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Catalog/"/>
        <thredds:catalogRef name="EarthProbe_TOMS_Level3" xlink:href="EarthProbe_TOMS_Level3/catalog.xml"
                      xlink:title="EarthProbe_TOMS_Level3"
                      xlink:type="simple"
                      ID="/opendap/hyrax/EarthProbe_TOMS_Level3/"/>
        <thredds:catalogRef name="HDF-EOS5" xlink:href="HDF-EOS5/catalog.xml" xlink:title="HDF-EOS5"
                      xlink:type="simple"
                      ID="/opendap/hyrax/HDF-EOS5/"/>
        <thredds:dataset name="HDF-EOS5.README" ID="/opendap/hyrax/HDF-EOS5.README">
     <thredds:dataSize units="bytes">242</thredds:dataSize>
     <thredds:date type="modified">2010-12-16T16:21:40</thredds:date>
     <thredds:access serviceName="file" urlPath="/HDF-EOS5.README"/>
  </thredds:dataset>
        <thredds:catalogRef name="Meteor3_TOMS_Level3" xlink:href="Meteor3_TOMS_Level3/catalog.xml"
                      xlink:title="Meteor3_TOMS_Level3"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Meteor3_TOMS_Level3/"/>
        <thredds:catalogRef name="Nimbus7_TOMS_Level3" xlink:href="Nimbus7_TOMS_Level3/catalog.xml"
                      xlink:title="Nimbus7_TOMS_Level3"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Nimbus7_TOMS_Level3/"/>
        <thredds:catalogRef name="Terra_MODIS_Level3" xlink:href="Terra_MODIS_Level3/catalog.xml"
                      xlink:title="Terra_MODIS_Level3"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Terra_MODIS_Level3/"/>
        <thredds:catalogRef name="ncml" xlink:href="ncml/catalog.xml" xlink:title="ncml"
                      xlink:type="simple"
                      ID="/opendap/hyrax/ncml/"/>
    </thredds:dataset>
</thredds:catalog>

http://acdisc.gsfc.nasa.gov/opendap/hyrax/Nimbus7_TOMS_Level3/catalog.xml

same drop and append structure

<thredds:catalog xmlns:fn="http://www.w3.org/2005/02/xpath-functions" xmlns:thredds="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:bes="http://xml.opendap.org/ns/bes/1.0#">
    <thredds:service name="dap" serviceType="OPeNDAP" base="/opendap/hyrax"/>
    <thredds:service name="file" serviceType="HTTPServer" base="/opendap/hyrax"/>
    <thredds:dataset name="/Nimbus7_TOMS_Level3" ID="/opendap/hyrax/Nimbus7_TOMS_Level3/">
        <thredds:catalogRef name="TOMSN7L3.008" xlink:href="TOMSN7L3.008/catalog.xml" xlink:title="TOMSN7L3.008" xlink:type="simple" ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3.008/"/>
        <thredds:catalogRef name="TOMSN7L3daer.008" xlink:href="TOMSN7L3daer.008/catalog.xml" xlink:title="TOMSN7L3daer.008" xlink:type="simple" ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3daer.008/"/>
        <thredds:catalogRef name="TOMSN7L3dery.008" xlink:href="TOMSN7L3dery.008/catalog.xml" xlink:title="TOMSN7L3dery.008" xlink:type="simple" ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/"/>
        <thredds:catalogRef name="TOMSN7L3dref.008" xlink:href="TOMSN7L3dref.008/catalog.xml" xlink:title="TOMSN7L3dref.008" xlink:type="simple" ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dref.008/"/>
        <thredds:catalogRef name="TOMSN7L3dtoz.008" xlink:href="TOMSN7L3dtoz.008/catalog.xml" xlink:title="TOMSN7L3dtoz.008" xlink:type="simple" ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dtoz.008/"/>
        <thredds:catalogRef name="TOMSN7L3maer.008" xlink:href="TOMSN7L3maer.008/catalog.xml" xlink:title="TOMSN7L3maer.008" xlink:type="simple" ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3maer.008/"/>
        <thredds:catalogRef name="TOMSN7L3mery.008" xlink:href="TOMSN7L3mery.008/catalog.xml" xlink:title="TOMSN7L3mery.008" xlink:type="simple" ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3mery.008/"/>
        <thredds:catalogRef name="TOMSN7L3mref.008" xlink:href="TOMSN7L3mref.008/catalog.xml" xlink:title="TOMSN7L3mref.008" xlink:type="simple" ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3mref.008/"/>
        <thredds:catalogRef name="TOMSN7L3mtoz.008" xlink:href="TOMSN7L3mtoz.008/catalog.xml" xlink:title="TOMSN7L3mtoz.008" xlink:type="simple" ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3mtoz.008/"/>
        <thredds:catalogRef name="TOMSN7L3zaer.008" xlink:href="TOMSN7L3zaer.008/catalog.xml" xlink:title="TOMSN7L3zaer.008" xlink:type="simple" ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3zaer.008/"/>
        <thredds:catalogRef name="TOMSN7L3zref.008" xlink:href="TOMSN7L3zref.008/catalog.xml" xlink:title="TOMSN7L3zref.008" xlink:type="simple" ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3zref.008/"/>
        <thredds:catalogRef name="TOMSN7L3ztoz.008" xlink:href="TOMSN7L3ztoz.008/catalog.xml" xlink:title="TOMSN7L3ztoz.008" xlink:type="simple" ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3ztoz.008/"/>
        <thredds:catalogRef name="TOMSN7OVP.008" xlink:href="TOMSN7OVP.008/catalog.xml" xlink:title="TOMSN7OVP.008" xlink:type="simple" ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7OVP.008/"/>
    </thredds:dataset>
</thredds:catalog>

http://acdisc.gsfc.nasa.gov/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/catalog.xml

seems stable across catalogRef elements

<?xml version="1.0" encoding="UTF-8"?>
<thredds:catalog xmlns:fn="http://www.w3.org/2005/02/xpath-functions"
             xmlns:thredds="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
             xmlns:xlink="http://www.w3.org/1999/xlink"
             xmlns:bes="http://xml.opendap.org/ns/bes/1.0#">
<thredds:service name="dap" serviceType="OPeNDAP" base="/opendap/hyrax"/>
<thredds:service name="file" serviceType="HTTPServer" base="/opendap/hyrax"/>
    <thredds:dataset name="/Nimbus7_TOMS_Level3/TOMSN7L3dery.008"
                ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/">
        <thredds:catalogRef name="1978" xlink:href="1978/catalog.xml" xlink:title="1978"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1978/"/>
        <thredds:catalogRef name="1979" xlink:href="1979/catalog.xml" xlink:title="1979"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1979/"/>
        <thredds:catalogRef name="1980" xlink:href="1980/catalog.xml" xlink:title="1980"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1980/"/>
        <thredds:catalogRef name="1981" xlink:href="1981/catalog.xml" xlink:title="1981"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1981/"/>
        <thredds:catalogRef name="1982" xlink:href="1982/catalog.xml" xlink:title="1982"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1982/"/>
        <thredds:catalogRef name="1983" xlink:href="1983/catalog.xml" xlink:title="1983"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1983/"/>
        <thredds:catalogRef name="1984" xlink:href="1984/catalog.xml" xlink:title="1984"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1984/"/>
        <thredds:catalogRef name="1985" xlink:href="1985/catalog.xml" xlink:title="1985"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1985/"/>
        <thredds:catalogRef name="1986" xlink:href="1986/catalog.xml" xlink:title="1986"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1986/"/>
        <thredds:catalogRef name="1987" xlink:href="1987/catalog.xml" xlink:title="1987"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1987/"/>
        <thredds:catalogRef name="1988" xlink:href="1988/catalog.xml" xlink:title="1988"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1988/"/>
        <thredds:catalogRef name="1989" xlink:href="1989/catalog.xml" xlink:title="1989"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1989/"/>
        <thredds:catalogRef name="1990" xlink:href="1990/catalog.xml" xlink:title="1990"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1990/"/>
        <thredds:catalogRef name="1991" xlink:href="1991/catalog.xml" xlink:title="1991"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1991/"/>
        <thredds:catalogRef name="1992" xlink:href="1992/catalog.xml" xlink:title="1992"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1992/"/>
        <thredds:catalogRef name="1993" xlink:href="1993/catalog.xml" xlink:title="1993"
                      xlink:type="simple"
                      ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1993/"/>
    </thredds:dataset>
</thredds:catalog>

http://acdisc.gsfc.nasa.gov/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1981/catalog.xml

at the dataset leaf node, goes back to the host + service/@base + dataset/@urlPath

note the @serviceName reference

<?xml version="1.0" encoding="UTF-8"?>
<thredds:catalog xmlns:fn="http://www.w3.org/2005/02/xpath-functions"
             xmlns:thredds="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
             xmlns:xlink="http://www.w3.org/1999/xlink"
             xmlns:bes="http://xml.opendap.org/ns/bes/1.0#">
    <thredds:service name="dap" serviceType="OPeNDAP" base="/opendap/hyrax"/>
    <thredds:service name="file" serviceType="HTTPServer" base="/opendap/hyrax"/>
    <thredds:dataset name="/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1981"
                ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1981/">
        <thredds:dataset name="L3_eryln_n7t_19810101.txt"
                           ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1981/L3_eryln_n7t_19810101.txt">
             <thredds:dataSize units="bytes">162783</thredds:dataSize>
             <thredds:date type="modified">2011-02-24T15:15:35</thredds:date>
             <thredds:access serviceName="file"
                             urlPath="/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1981/L3_eryln_n7t_19810101.txt"/>
          </thredds:dataset>
        <thredds:dataset name="L3_eryln_n7t_19810102.txt"
                   ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1981/L3_eryln_n7t_19810102.txt">
             <thredds:dataSize units="bytes">162783</thredds:dataSize>
             <thredds:date type="modified">2011-02-24T15:15:35</thredds:date>
             <thredds:access serviceName="file"
                             urlPath="/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1981/L3_eryln_n7t_19810102.txt"/>
          </thredds:dataset>
        <thredds:dataset name="L3_eryln_n7t_19810103.txt"
                   ID="/opendap/hyrax/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1981/L3_eryln_n7t_19810103.txt">
             <thredds:dataSize units="bytes">162783</thredds:dataSize>
             <thredds:date type="modified">2011-02-24T15:15:36</thredds:date>
             <thredds:access serviceName="file"
                             urlPath="/Nimbus7_TOMS_Level3/TOMSN7L3dery.008/1981/L3_eryln_n7t_19810103.txt"/>
          </thredds:dataset>
     </thredds:dataset>
</thredds:catalog>

4. NKN v 1.0.1

URL: http://thredds.northwestknowledge.net:8080/thredds/catalog.xml

The tree.

http://thredds.northwestknowledge.net:8080/thredds/catalog.xml

service block

drop & append structure (urljoin)

<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" name="Northwest Knowledge Network (NKN) - THREDDS Server" version="1.0.1">
  <service name="all" serviceType="Compound" base="">
    <service name="odap" serviceType="OPENDAP" base="/thredds/dodsC/" />
    <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
    <service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/grid/" />
  </service>
  <catalogRef xlink:href="/thredds/nkn.data.xml" xlink:title="Northwest Knowledge Network (NKN) Data" name="" />
  <catalogRef xlink:href="/thredds/nw.csc.is.catalog.xml" xlink:title="USGS NW Climate Science Center Integrated Scenarios" name="" />
</catalog>

http://thredds.northwestknowledge.net:8080/thredds/nkn.data.xml

host + catalogRef/@urlPath


<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" name="Northwest Knowledge Network (NKN) - THREDDS Catalog" version="1.0.1">
  <service name="all" serviceType="Compound" base="">
    <service name="odap" serviceType="OPENDAP" base="/thredds/dodsC/" />
    <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
    <service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/grid/" />
    <service name="iso" serviceType="ISO" base="/thredds/iso/" />
  </service>
  <catalogRef name="" harvest="true" ID="NKN_DATA_ALL_SCAN" xlink:href="/thredds/catalog/NKN_PUBLISHED_DATA/catalog.xml" xlink:title="NKN Published Data">
    <metadata inherited="true">
      <serviceName>all</serviceName>
      <dataType>GRID</dataType>
    </metadata>
    <property name="DatasetScan" value="true" />
  </catalogRef>
</catalog>

http://thredds.northwestknowledge.net:8080/thredds/catalog/NKN_PUBLISHED_DATA/catalog.xml

and it's just uuids all the way down (separate issue - how can anyone know what they're looking through?)

urljoin


<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.0.1">
  <service name="all" serviceType="Compound" base="">
    <service name="odap" serviceType="OPENDAP" base="/thredds/dodsC/" />
    <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
    <service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/grid/" />
    <service name="iso" serviceType="ISO" base="/thredds/iso/" />
  </service>
  <dataset name="NKN Published Data" harvest="true" ID="NKN_DATA_ALL_SCAN">
    <metadata inherited="true">
      <serviceName>all</serviceName>
      <dataType>GRID</dataType>
    </metadata>
    <catalogRef xlink:href="f6e1ae8e-4a26-473b-9dd7-0031cc5720db/catalog.xml" xlink:title="f6e1ae8e-4a26-473b-9dd7-0031cc5720db" ID="NKN_DATA_ALL_SCAN/f6e1ae8e-4a26-473b-9dd7-0031cc5720db" name="" />
    <catalogRef xlink:href="d986d1f3-89c1-4c76-9946-eb750f3e921a/catalog.xml" xlink:title="d986d1f3-89c1-4c76-9946-eb750f3e921a" ID="NKN_DATA_ALL_SCAN/d986d1f3-89c1-4c76-9946-eb750f3e921a" name="" />
    <catalogRef xlink:href="d612d94b-df2b-4181-aa6e-b4513c0ce88f/catalog.xml" xlink:title="d612d94b-df2b-4181-aa6e-b4513c0ce88f" ID="NKN_DATA_ALL_SCAN/d612d94b-df2b-4181-aa6e-b4513c0ce88f" name="" />
  </dataset>
</catalog>

http://thredds.northwestknowledge.net:8080/thredds/catalog/NKN_PUBLISHED_DATA/f6e1ae8e-4a26-473b-9dd7-0031cc5720db/catalog.xml


<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.0.1">
  <service name="all" serviceType="Compound" base="">
    <service name="odap" serviceType="OPENDAP" base="/thredds/dodsC/" />
    <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
    <service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/grid/" />
    <service name="iso" serviceType="ISO" base="/thredds/iso/" />
  </service>
  <dataset name="f6e1ae8e-4a26-473b-9dd7-0031cc5720db" ID="NKN_DATA_ALL_SCAN/f6e1ae8e-4a26-473b-9dd7-0031cc5720db">
    <metadata inherited="true">
      <serviceName>all</serviceName>
      <dataType>GRID</dataType>
    </metadata>
    <catalogRef xlink:href="fff46db4-b1eb-4c5a-bce5-7a2a39af82ce/catalog.xml" xlink:title="fff46db4-b1eb-4c5a-bce5-7a2a39af82ce" ID="NKN_DATA_ALL_SCAN/f6e1ae8e-4a26-473b-9dd7-0031cc5720db/fff46db4-b1eb-4c5a-bce5-7a2a39af82ce" name="" />
    <catalogRef xlink:href="fff06b00-28da-4ef6-b97c-e8094d7accb0/catalog.xml" xlink:title="fff06b00-28da-4ef6-b97c-e8094d7accb0" ID="NKN_DATA_ALL_SCAN/f6e1ae8e-4a26-473b-9dd7-0031cc5720db/fff06b00-28da-4ef6-b97c-e8094d7accb0" name="" />
    <catalogRef xlink:href="ffeefbb8-6dfe-4ad5-b96f-4c049ae0c439/catalog.xml" xlink:title="ffeefbb8-6dfe-4ad5-b96f-4c049ae0c439" ID="NKN_DATA_ALL_SCAN/f6e1ae8e-4a26-473b-9dd7-0031cc5720db/ffeefbb8-6dfe-4ad5-b96f-4c049ae0c439" name="" />
    <catalogRef xlink:href="ffe33356-bd34-4be2-be3a-8d7260793a02/catalog.xml" xlink:title="ffe33356-bd34-4be2-be3a-8d7260793a02" ID="NKN_DATA_ALL_SCAN/f6e1ae8e-4a26-473b-9dd7-0031cc5720db/ffe33356-bd34-4be2-be3a-8d7260793a02" name="" />
    <catalogRef xlink:href="ffdec6c9-a149-4c20-b53f-540e993970d8/catalog.xml" xlink:title="ffdec6c9-a149-4c20-b53f-540e993970d8" ID="NKN_DATA_ALL_SCAN/f6e1ae8e-4a26-473b-9dd7-0031cc5720db/ffdec6c9-a149-4c20-b53f-540e993970d8" name="" />
    <catalogRef xlink:href="ffd238fb-7687-48f9-a3c9-c8031fef71ec/catalog.xml" xlink:title="ffd238fb-7687-48f9-a3c9-c8031fef71ec" ID="NKN_DATA_ALL_SCAN/f6e1ae8e-4a26-473b-9dd7-0031cc5720db/ffd238fb-7687-48f9-a3c9-c8031fef71ec" name="" />
  </dataset>
</catalog>

http://thredds.northwestknowledge.net:8080/thredds/catalog/NKN_PUBLISHED_DATA/f6e1ae8e-4a26-473b-9dd7-0031cc5720db/fff46db4-b1eb-4c5a-bce5-7a2a39af82ce/catalog.xml

no urlPath (this is a bad dataset choice, I think)

<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.0.1">
  <service name="all" serviceType="Compound" base="">
    <service name="odap" serviceType="OPENDAP" base="/thredds/dodsC/" />
    <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
    <service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/grid/" />
    <service name="iso" serviceType="ISO" base="/thredds/iso/" />
  </service>
  <dataset name="fff46db4-b1eb-4c5a-bce5-7a2a39af82ce" ID="NKN_DATA_ALL_SCAN/f6e1ae8e-4a26-473b-9dd7-0031cc5720db/fff46db4-b1eb-4c5a-bce5-7a2a39af82ce">
    <metadata inherited="true">
      <serviceName>all</serviceName>
      <dataType>GRID</dataType>
    </metadata>
  </dataset>
</catalog>

http://thredds.northwestknowledge.net:8080/thredds/catalog/NKN_PUBLISHED_DATA/D355B53A-EC40-472F-919C-D822578FA316/FFE4FB86-FA99-4634-9C9F-BFA1FE580FB6/catalog.xml

valid dataset leaf node

note the change in casing between the catalog.xmls - different dataset configurations?

host + service/@base + dataset/@urlPath

note that despite so many uuids, the iso record does not include a fileIdentifier or dataidentification/identifier

dang it, ed, codelists just aren't that complicated: http://example.org/someUnitsDictionary.xml#degrees_east

<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.0.1">
  <service name="all" serviceType="Compound" base="">
    <service name="odap" serviceType="OPENDAP" base="/thredds/dodsC/" />
    <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
    <service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/grid/" />
    <service name="iso" serviceType="ISO" base="/thredds/iso/" />
  </service>
  <dataset name="FFE4FB86-FA99-4634-9C9F-BFA1FE580FB6" ID="NKN_DATA_ALL_SCAN/D355B53A-EC40-472F-919C-D822578FA316/FFE4FB86-FA99-4634-9C9F-BFA1FE580FB6">
    <metadata inherited="true">
      <serviceName>all</serviceName>
      <dataType>GRID</dataType>
    </metadata>
    <dataset name="dcsdelta_MPI-M_ECHAM5-OM_A1B_20102099_tmin_run2_snake_epscor.nc" ID="NKN_DATA_ALL_SCAN/D355B53A-EC40-472F-919C-D822578FA316/FFE4FB86-FA99-4634-9C9F-BFA1FE580FB6/dcsdelta_MPI-M_ECHAM5-OM_A1B_20102099_tmin_run2_snake_epscor.nc" urlPath="NKN_PUBLISHED_DATA/D355B53A-EC40-472F-919C-D822578FA316/FFE4FB86-FA99-4634-9C9F-BFA1FE580FB6/dcsdelta_MPI-M_ECHAM5-OM_A1B_20102099_tmin_run2_snake_epscor.nc">
      <dataSize units="Mbytes">447.4</dataSize>
      <date type="modified">2013-11-16T08:49:57Z</date>
    </dataset>
  </dataset>
</catalog>

5. Ferret v1.0.1

URL: http://ferret.pmel.noaa.gov/thredds/catalog/catalog.xml

The tree.

http://ferret.pmel.noaa.gov/thredds/catalog/catalog.xml

compound service blocks

catalogRef/@href - host + append

additional catalogRef/serviceName to id which set of services supported

<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" name="PMEL THREDDS server" version="1.0.1">
  <service name="all" serviceType="Compound" base="">
    <service name="odap" serviceType="OPENDAP" base="/thredds/dodsC/" />
    <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
  </service>
  <service name="gridded" serviceType="Compound" base="">
    <service name="opendap" serviceType="OPENDAP" base="/thredds/dodsC/" />
    <service name="wcs" serviceType="WCS" base="/thredds/wcs/" />
  </service>
  <catalogRef name="" xlink:href="/thredds/catalog/las/catalog.xml" xlink:title="Data From NVODS">
    <serviceName>gridded</serviceName>
    <property name="DatasetScan" value="true" />
  </catalogRef>
  <catalogRef xlink:href="carbontracker.xml" xlink:title="CarbonTracker CT2009" ID="carbonTracker" name="" />
  <catalogRef xlink:href="pmel_catalog.xml" xlink:title="PMEL" ID="pmel_cat" name="" />
</catalog>

http://ferret.pmel.noaa.gov/thredds/catalog/las/catalog.xml

dataset/serviceName (instead of dataset/@serviceName)

back to urljoin href (replace catalog.xml with @href)

<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.0.1">
    <service name="gridded" serviceType="Compound" base="">
        <service name="opendap" serviceType="OPENDAP" base="/thredds/dodsC/"/>
        <service name="wcs" serviceType="WCS" base="/thredds/wcs/"/>
    </service>
    <dataset name="Data From NVODS" ID="las">
        <serviceName>gridded</serviceName>
        <metadata inherited="true">
            <serviceName>gridded</serviceName>
        </metadata>
        <catalogRef xlink:href="woa05_seasonal/catalog.xml" xlink:title="woa05_seasonal" ID="las/woa05_seasonal" name=""/>
        <catalogRef xlink:href="woa05_monthly/catalog.xml" xlink:title="woa05_monthly" ID="las/woa05_monthly" name=""/>
        <catalogRef xlink:href="woa05_annual/catalog.xml" xlink:title="woa05_annual" ID="las/woa05_annual" name=""/>
        <catalogRef xlink:href="woa01_seasonal/catalog.xml" xlink:title="woa01_seasonal" ID="las/woa01_seasonal" name=""/>
        <catalogRef xlink:href="woa01_monthly/catalog.xml" xlink:title="woa01_monthly" ID="las/woa01_monthly" name=""/>
        <catalogRef xlink:href="woa01_annual/catalog.xml" xlink:title="woa01_annual" ID="las/woa01_annual" name=""/>
        <catalogRef xlink:href="tmi_filled_one_weekly/catalog.xml" xlink:title="tmi_filled_one_weekly" ID="las/tmi_filled_one_weekly" name=""/>
        <catalogRef xlink:href="tmi_filled_one_seasonal/catalog.xml" xlink:title="tmi_filled_one_seasonal" ID="las/tmi_filled_one_seasonal" name=""/>
    </dataset>
</catalog>

http://ferret.pmel.noaa.gov/thredds/catalog/las/woa05_annual/catalog.xml

host + service base + href

serviceName is at the parent dataset bucket level

also a functional wcs service

<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.0.1">
    <service name="gridded" serviceType="Compound" base="">
        <service name="opendap" serviceType="OPENDAP" base="/thredds/dodsC/" />
        <service name="wcs" serviceType="WCS" base="/thredds/wcs/" />
    </service>
    <dataset name="woa05_annual" ID="las/woa05_annual">
        <serviceName>gridded</serviceName>
        <metadata inherited="true">
            <serviceName>gridded</serviceName>
        </metadata>
        <dataset name="data_ferret.pmel.noaa.gov_thredds_dodsC_data_PMEL_WOA05nc_annual_t00se1.nc.jnl" ID="las/woa05_annual/data_ferret.pmel.noaa.gov_thredds_dodsC_data_PMEL_WOA05nc_annual_t00se1.nc.jnl" urlPath="las/woa05_annual/data_ferret.pmel.noaa.gov_thredds_dodsC_data_PMEL_WOA05nc_annual_t00se1.nc.jnl">
            <dataSize units="bytes">84.0</dataSize>
            <date type="modified">2015-01-29T00:05:52Z</date>
        </dataset>
        <dataset name="data_ferret.pmel.noaa.gov_thredds_dodsC_data_PMEL_WOA05nc_annual_t00sd1.nc.jnl" ID="las/woa05_annual/data_ferret.pmel.noaa.gov_thredds_dodsC_data_PMEL_WOA05nc_annual_t00sd1.nc.jnl" urlPath="las/woa05_annual/data_ferret.pmel.noaa.gov_thredds_dodsC_data_PMEL_WOA05nc_annual_t00sd1.nc.jnl">
            <dataSize units="bytes">84.0</dataSize>
            <date type="modified">2015-01-29T00:05:52Z</date>
        </dataset>
        <dataset name="data_ferret.pmel.noaa.gov_thredds_dodsC_data_PMEL_WOA05nc_annual_t00mn1.nc.jnl" ID="las/woa05_annual/data_ferret.pmel.noaa.gov_thredds_dodsC_data_PMEL_WOA05nc_annual_t00mn1.nc.jnl" urlPath="las/woa05_annual/data_ferret.pmel.noaa.gov_thredds_dodsC_data_PMEL_WOA05nc_annual_t00mn1.nc.jnl">
            <dataSize units="bytes">84.0</dataSize>
            <date type="modified">2015-01-29T00:05:53Z</date>
        </dataset>
        <dataset name="data_ferret.pmel.noaa.gov_thredds_dodsC_data_PMEL_WOA05nc_annual_t00ma1.nc.jnl" ID="las/woa05_annual/data_ferret.pmel.noaa.gov_thredds_dodsC_data_PMEL_WOA05nc_annual_t00ma1.nc.jnl" urlPath="las/woa05_annual/data_ferret.pmel.noaa.gov_thredds_dodsC_data_PMEL_WOA05nc_annual_t00ma1.nc.jnl">
            <dataSize units="bytes">84.0</dataSize>
            <date type="modified">2015-01-29T00:05:51Z</date>
        </dataset>
    </dataset>
</catalog>

6. PML v1.0.1

URL: https://rsg.pml.ac.uk/thredds/catalog/catalog.xml

The tree.

https://rsg.pml.ac.uk/thredds/catalog/catalog.xml

urljoin (replace catalog.xml with catalogRef/@href)

note the "non-standard" catalog.xml name

<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" name="PML Remote Sensing Group operational THREDDS Server" version="1.0.1">
    <service name="all" serviceType="Compound" base="">
        <service name="odap" serviceType="OPENDAP" base="/thredds/dodsC/" />
        <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
        <service name="wcs" serviceType="WCS" base="/thredds/wcs/" />
        <service name="wms" serviceType="WMS" base="/thredds/wms/" />
        <service name="ncml" serviceType="NCML" base="/thredds/ncml/" />
        <service name="uddc" serviceType="Data discovery report" base="/thredds/uddc/" />
        <service name="iso" serviceType="ISO19115/Inspire" base="/thredds/iso/" />
    </service>
    <catalogRef xlink:href="catalog-earth2observe.xml" xlink:title="EartH2Observe" name="" />
    <catalogRef xlink:href="catalog-metoffice.xml" xlink:title="Met Office MCRS" name="" />
    <catalogRef xlink:href="catalog-cci.xml" xlink:title="Ocean Colour - CCI" name="" />
    <catalogRef xlink:href="catalog-opec.xml" xlink:title="Operational Ecology" name="" />
    <catalogRef xlink:href="catalog-twap.xml" xlink:title="TWAP" name="" />
</catalog>

https://rsg.pml.ac.uk/thredds/catalog/catalog-earth2observe.xml

metadata! with actual metadata!

dataset bucket with catalogRef and leaf datasets (but no data size, etc, info)

rel paths include the catalog route piece (host + append @catalogRef/@href

<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
    xmlns:xlink="http://www.w3.org/1999/xlink"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0 http://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.0.1.xsd"
    name="PML RSG Web Services eartH2Observe project catalogue" version="1.0.1">
    <service name="all" serviceType="Compound" base="">
        <service name="odap" serviceType="OPENDAP" base="/thredds/dodsC/"/>
        <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/"/>
        <service name="wcs" serviceType="WCS" base="/thredds/wcs/"/>
        <service name="wms" serviceType="WMS" base="/thredds/wms/"/>
        <service name="ncml" serviceType="NCML" base="/thredds/ncml/"/>
        <service name="uddc" serviceType="Data discovery report" base="/thredds/uddc/"/>
        <service name="iso" serviceType="ISO19115/Inspire" base="/thredds/iso/"/>
    </service>
    <dataset name="DATA">
        <dataset name="CEH">
            <metadata inherited="true">
                <serviceName>all</serviceName>
                <authority>ceh.ac.uk</authority>
                <dataType>GRID</dataType>
                <creator>
                    <name vocabulary="DIF">Alberto Martinez de La Torre</name>
                    <contact email="albmar@ceh.ac.uk"/>
                </creator>
            </metadata>
            <catalogRef name="" ID="ceh-wrr1" xlink:href="/thredds/catalog/ceh/wrr1/catalog.xml"
                xlink:title="CEH Water Resource Re-analysis v1">
                <property name="DatasetScan" value="true"/>
            </catalogRef>
            <dataset name="CEH Water Resource Re-analysis v1 - daily (aggregated)"
                ID="ceh-wrr1-daily-agg" urlPath="ceh/wrr1-daily-agg.nc"/>
            <dataset name="CEH Water Resource Re-analysis v1 - monthly (aggregated)"
                ID="ceh-wrr1-monthly-agg" urlPath="ceh/wrr1-monthly-agg.nc"/>
        </dataset>
        <dataset name="CNR">
            <metadata inherited="true">
                <serviceName>all</serviceName>
                <authority>isac.cnr.it</authority>
                <dataType>GRID</dataType>
            </metadata>
            <dataset name="3B42">
                <metadata inherited="true">
                    <documentation type="Summary"/>
                    <documentation type="Rights">Both products are freely downloadable from the
                        Goddard Earth Sciences Data and Information Services Center (GES-DISC). GES
                        DISC Data Use Acknowledgment (from
                        http://disc.sci.gsfc.nasa.gov/additional/citing-our-data) Distribution of
                        GES DISC data sets is funded by NASA's Science Mission Directorate (SMD).
                        The data are not copyrighted and are open to all for both commercial and
                        non-commercial uses. If you used GES DISC data for a publication (research
                        or otherwise), or for any other purpose, we request that you include the
                        following acknowledgment: "The data used in this effort were acquired as
                        part of the activities of NASA's Science Mission Directorate, and are
                        archived and distributed by the Goddard Earth Sciences (GES) Data and
                        Information Services Center (DISC)." We would appreciate receiving a copy of
                        your publication, which can be be forwarded to the following address:
                        Goddard Earth Sciences DISC Help Desk Code 610.2 NASA/Goddard Space Flight
                        Center Greenbelt, MD 20771 Or, you may email the publication reference to
                        gsfc-help-disc@lists.nasa.gov.</documentation>
                    <creator>
                        <name vocabulary="DIF">Elsa Cattani (CNR)</name>
                        <contact url="http://www.cnr.it/sitocnr/ServiziSito/contattaci_eng.html"
                            email="e.cattani@isac.cnr.it"/>
                    </creator>
                </metadata>
                <catalogRef name="" ID="cnr-3b42-rainfall-3h"
                    xlink:href="/thredds/catalog/cnr/3b42-3h/catalog.xml"
                    xlink:title="3B42 3-Hourly Rainfall Total">
                    <property name="DatasetScan" value="true"/>
                </catalogRef>
                <dataset name="3B42 3-Hourly Rainfall Total (aggregated)"
                    ID="cnr-3b42-rainfall-3h-agg" urlPath="cnr/3b42-rainfall-3h-agg.nc"/>
                <dataset name="3B42 3-Hourly Rainfall Total (aggregated), annual files"
                    ID="cnr-3b42-rainfall-3h-annual-agg"
                    urlPath="cnr/3b42-rainfall-3h-annual-agg.nc"/>
                <catalogRef name="" ID="cnr-3b42-rainfall-daily"
                    xlink:href="/thredds/catalog/cnr/3b42-daily/catalog.xml"
                    xlink:title="3B42 Daily Rainfall Total">
                    <property name="DatasetScan" value="true"/>
                </catalogRef>
                <dataset name="3B42 Daily Rainfall Total (aggregated)"
                    ID="cnr-3b42-rainfall-daily-agg" urlPath="cnr/3b42-rainfall-daily-agg.nc"/>
            </dataset>
            <dataset name="CMORPH">
                <metadata inherited="true">
                    <documentation type="Summary"/>
                    <documentation type="Rights"/>
                    <creator>
                        <name vocabulary="DIF">Elsa Cattani (CNR)</name>
                        <contact url="http://www.cnr.it/sitocnr/ServiziSito/contattaci_eng.html"
                            email="e.cattani@isac.cnr.it"/>
                    </creator>
                </metadata>
                <catalogRef name="" ID="cnr-cmorph-raintotal-daily"
                    xlink:href="/thredds/catalog/cnr/cmorph-daily/catalog.xml"
                    xlink:title="CMORPH Daily Rainfall Total">
                    <property name="DatasetScan" value="true"/>
                </catalogRef>
                <dataset name="CMORPH Daily Rainfall Total (aggregated)"
                    ID="cnr-cmorph-rainfall-daily-agg" urlPath="cnr/cmorph-daily-rainfall-agg.nc"/>
            </dataset>
            <dataset name="TAMSAT">
                <metadata inherited="true">
                    <documentation type="Summary"/>
                    <documentation type="Rights">By downloading the TAMSAT rainfall estimates or the
                        TARCAT dataset, you are agreeing to the following terms of use, which have
                        been established considering the rules of the various funding bodies and
                        collaborating parties, under our past and present contracts and agreements:
                        - The TAMSAT rainfall estimation system and the TARCAT dataset (henceforth
                        referred to as TAMSAT/TARCAT) are owned by the University of Reading. - Use
                        of TAMSAT/TARCAT for any purpose is at the User's own risk. Users accept
                        that the University of Reading is not liable for the consequences of using
                        TAMSAT/TARCAT data, for any mistakes in the data or for any discrepancies
                        with other observed datasets. - TAMSAT/TARCAT can be freely used for
                        research purposes, as long as Tarnavsky et al, 2014 and Maidment et al, 2014
                        are acknowledged in any resulting publications or presentations. - Any
                        operational or commercial use of TAMSAT must be explicitly agreed by the
                        University of Reading. - Any operational or commercial re-distribution of
                        TAMSAT/TARCAT, including images derived from TAMSAT/TARCAT data or
                        descriptions based on TAMSAT/TARCAT data must be explicitly agreed with the
                        University of Reading. The following are representative examples of such
                        usage: the inclusion of TARCAT based images in famine early warning
                        bulletins; the use of TARCAT data in risk management/assessment tools; the
                        use of TARCAT data for operating weather index insurance; Tarnavsky, E., D.
                        Grimes, R. Maidment, E. Black, R. Allan, M. Stringer, R. Chadwick, F.
                        Kayitakire (2014) Extension of the TAMSAT Satellite-based Rainfall
                        Monitoring over Africa and from 1983 to present Journal of Applied
                        Meteorology and Climate DOI 10.1175/JAMC-D-14-0016.1 Maidment, R., D.
                        Grimes, R.P.Allan, E. Tarnavsky, M. Stringer, T. Hewison, R. Roebeling and
                        E. Black (2014) The 30 year TAMSAT African Rainfall Climatology And Time
                        series (TARCAT) data set Journal of Geophysical Research DOI:
                        10.1002/2014JD021927</documentation>
                    <creator>
                        <name vocabulary="DIF">Elsa Cattani (CNR)</name>
                        <contact url="http://www.cnr.it/sitocnr/ServiziSito/contattaci_eng.html"
                            email="e.cattani@isac.cnr.it"/>
                    </creator>
                </metadata>
                <catalogRef name="" ID="cnr-tamsat-rfe-dekadal"
                    xlink:href="/thredds/catalog/cnr/tamsat-rfe-dekadal/catalog.xml"
                    xlink:title="TAMSAT Rainfall Estimate Dekadal">
                    <property name="DatasetScan" value="true"/>
                </catalogRef>
                <dataset name="TAMSAT Rainfall Estimate Dekadal (aggregated)"
                    ID="cnr-rfe-dekadal-agg" urlPath="cnr/tamsat-rfe-dekadal-agg.nc"/>
                <catalogRef name="" ID="cnr-tamsat-rfe-monthly"
                    xlink:href="/thredds/catalog/cnr/tamsat-rfe-monthly/catalog.xml"
                    xlink:title="TAMSAT Rainfall Estimate Monthly">
                    <property name="DatasetScan" value="true"/>
                </catalogRef>
                <dataset name="TAMSAT Rainfall Estimate Monthly (aggregated)"
                    ID="cnr-tamsat-rfe-monthly-agg" urlPath="cnr/tamsat-rfe-monthly-agg.nc"/>
            </dataset>
        </dataset>
    </dataset>
</catalog>

https://rsg.pml.ac.uk/thredds/catalog/cnr/3b42-3h/catalog.xml

(dataset/dataset/catalogRef)

inherited metadata that is also duplicated

<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.0.1">
    <service name="all" serviceType="Compound" base="">
        <service name="odap" serviceType="OPENDAP" base="/thredds/dodsC/" />
        <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
        <service name="wcs" serviceType="WCS" base="/thredds/wcs/" />
        <service name="wms" serviceType="WMS" base="/thredds/wms/" />
        <service name="ncml" serviceType="NCML" base="/thredds/ncml/" />
        <service name="uddc" serviceType="Data discovery report" base="/thredds/uddc/" />
        <service name="iso" serviceType="ISO19115/Inspire" base="/thredds/iso/" />
    </service>
    <dataset name="3B42 3-Hourly Rainfall Total" ID="cnr-3b42-rainfall-3h">
        <metadata inherited="true">
            <serviceName>all</serviceName>
            <authority>isac.cnr.it</authority>
            <dataType>GRID</dataType>
            <documentation type="Summary" />
            <documentation type="Rights">Both products are freely downloadable from the Goddard Earth Sciences Data and Information Services Center (GES-DISC). GES DISC Data Use Acknowledgment (from http://disc.sci.gsfc.nasa.gov/additional/citing-our-data) Distribution of GES DISC data sets is funded by NASA's Science Mission Directorate (SMD). The data are not copyrighted and are open to all for both commercial and non-commercial uses. If you used GES DISC data for a publication (research or otherwise), or for any other purpose, we request that you include the following acknowledgment: "The data used in this effort were acquired as part of the activities of NASA's Science Mission Directorate, and are archived and distributed by the Goddard Earth Sciences (GES) Data and Information Services Center (DISC)." We would appreciate receiving a copy of your publication, which can be be forwarded to the following address: Goddard Earth Sciences DISC Help Desk Code 610.2 NASA/Goddard Space Flight Center Greenbelt, MD 20771 Or, you may email the publication reference to gsfc-help-disc@lists.nasa.gov.</documentation>
            <creator>
                <name vocabulary="DIF">Elsa Cattani (CNR)</name>
                <contact url="http://www.cnr.it/sitocnr/ServiziSito/contattaci_eng.html" email="e.cattani@isac.cnr.it" />
            </creator>
        </metadata>
        <catalogRef xlink:href="1998/catalog.xml" xlink:title="1998" ID="cnr-3b42-rainfall-3h/1998" name="" />
        <catalogRef xlink:href="1999/catalog.xml" xlink:title="1999" ID="cnr-3b42-rainfall-3h/1999" name="" />
        <catalogRef xlink:href="2000/catalog.xml" xlink:title="2000" ID="cnr-3b42-rainfall-3h/2000" name="" />
        <catalogRef xlink:href="annual/catalog.xml" xlink:title="annual" ID="cnr-3b42-rainfall-3h/annual" name="" />
    </dataset>
</catalog>

(tracing similar catalog structures, not including examples of each)

https://rsg.pml.ac.uk/thredds/catalog/cnr/3b42-3h/1998/catalog.xml

https://rsg.pml.ac.uk/thredds/catalog/cnr/3b42-3h/1998/01/catalog.xml

https://rsg.pml.ac.uk/thredds/catalog/cnr/3b42-3h/1998/01/01/catalog.xml

dataset now has the data size and date children

no parent-level or element-level service ref

metadata contains dataype (grid) element but never seen a dataset element that contained a type

<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.0.1">
    <service name="all" serviceType="Compound" base="">
        <service name="odap" serviceType="OPENDAP" base="/thredds/dodsC/" />
        <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
        <service name="wcs" serviceType="WCS" base="/thredds/wcs/" />
        <service name="wms" serviceType="WMS" base="/thredds/wms/" />
        <service name="ncml" serviceType="NCML" base="/thredds/ncml/" />
        <service name="uddc" serviceType="Data discovery report" base="/thredds/uddc/" />
        <service name="iso" serviceType="ISO19115/Inspire" base="/thredds/iso/" />
    </service>
    <dataset name="01" ID="cnr-3b42-rainfall-3h/1998/01/01">
        <metadata inherited="true">
            <serviceName>all</serviceName>
            <authority>isac.cnr.it</authority>
            <dataType>GRID</dataType>
            <documentation type="Summary" />
            <documentation type="Rights">Both products are freely downloadable from the Goddard Earth Sciences Data and Information Services Center (GES-DISC). GES DISC Data Use Acknowledgment (from http://disc.sci.gsfc.nasa.gov/additional/citing-our-data) Distribution of GES DISC data sets is funded by NASA's Science Mission Directorate (SMD). The data are not copyrighted and are open to all for both commercial and non-commercial uses. If you used GES DISC data for a publication (research or otherwise), or for any other purpose, we request that you include the following acknowledgment: "The data used in this effort were acquired as part of the activities of NASA's Science Mission Directorate, and are archived and distributed by the Goddard Earth Sciences (GES) Data and Information Services Center (DISC)." We would appreciate receiving a copy of your publication, which can be be forwarded to the following address: Goddard Earth Sciences DISC Help Desk Code 610.2 NASA/Goddard Space Flight Center Greenbelt, MD 20771 Or, you may email the publication reference to gsfc-help-disc@lists.nasa.gov.</documentation>
            <creator>
                <name vocabulary="DIF">Elsa Cattani (CNR)</name>
                <contact url="http://www.cnr.it/sitocnr/ServiziSito/contattaci_eng.html" email="e.cattani@isac.cnr.it" />
            </creator>
        </metadata>
        <dataset name="3B42.19980101.00.7.nc" ID="cnr-3b42-rainfall-3h/1998/01/01/3B42.19980101.00.7.nc" urlPath="cnr/3b42-3h/1998/01/01/3B42.19980101.00.7.nc">
            <dataSize units="Mbytes">13.84</dataSize>
            <date type="modified">2014-10-27T08:32:46Z</date>
        </dataset>
        <dataset name="3B42.19980101.03.7.nc" ID="cnr-3b42-rainfall-3h/1998/01/01/3B42.19980101.03.7.nc" urlPath="cnr/3b42-3h/1998/01/01/3B42.19980101.03.7.nc">
            <dataSize units="Mbytes">13.84</dataSize>
            <date type="modified">2014-10-27T08:32:46Z</date>
        </dataset>
    </dataset>
</catalog>

OPeNDAP Tidbits

http://opendap.co-ops.nos.noaa.gov/axis/webservices/waterlevelverifieddaily/samples/request.xml http://opendap.co-ops.nos.noaa.gov/axis/webservices/waterlevelverifieddaily/wsdl/

which read like OPeNDAP data access (parameter-dependent) through WSDL SOAP requests but the wsdl endpoints aren't valid from just reading the URL, no modifications.





In [ ]:

Content source: b-cube/thredds_catalog_crawler

Similar notebooks:

notebook.community | gallery | about